Skip to content

Bring the cloud (cloudbridge) object store to parity with the boto3 store#23098

Open
nuwang wants to merge 20 commits into
galaxyproject:devfrom
nuwang:cloud-objectstore-parity
Open

Bring the cloud (cloudbridge) object store to parity with the boto3 store#23098
nuwang wants to merge 20 commits into
galaxyproject:devfrom
nuwang:cloud-objectstore-parity

Conversation

@nuwang

@nuwang nuwang commented Jul 10, 2026

Copy link
Copy Markdown
Member

Brings lib/galaxy/objectstore/cloud.py to functional parity with the modern boto3 store, making it a viable multi-cloud successor ahead of eventually deprecating the legacy boto2 s3.py store. Builds on cloudbridge 4.3.0, which added cross-provider multipart uploads (CloudVE/cloudbridge#333), a Python 3.10 floor (#338), response-header overrides on signed URLs (#339), and ranged parallel downloads (#340).

What's included (one commit per change, tests first in each)

Bug fixes

  • AWS region on <auth> was silently dropped ("region" in auth_element iterates child elements, never attributes)
  • Missing GCP credentials_file raised TypeError instead of the missing-configuration error
  • Both push paths did a double objects.get() per upload
  • The store never declared cloud = True (badge serialization)
  • Listings only consumed the first page (cloudbridge's list() returns one page, default 50 items) — delete-all and directory-into-cache handled datasets with many extra files incompletely; they now page through the full result set via iter()
  • Test-helper bug: verify_caching_object_store_functionality's "bigger file" was written but never uploaded (and nothing was asserted)

Parity features

  • Transfer tuning in both directions: new <transfer> block (multipart_threshold, multipart_chunksize, max_concurrency — boto3-store naming, with optional upload_/download_ prefixes to tune one direction) mapped to cloudbridge's per-call TransferConfig. Upload chunk sizes below the 5 MiB provider minimum are rejected at construction (downloads have no minimum).
  • Ranged parallel downloads: _download_to now uses cloudbridge's download_to_file, which fetches large objects as parallel ranged reads — mirroring what boto3's TransferManager gives the boto3 store. The axel fallback is dropped (parallel downloads are native now; the boto3 store never used axel either).
  • Direct download: enable_direct_download now works for the cloud store — content_disposition/content_type are forwarded to generate_url (full support on AWS/Azure/GCP; Swift honors the filename portion via tempurl).
  • Full provider initialization surface: AWS session_token + <connection endpoint_url= validate_certs= signature_version=> (S3-compatible endpoints; the scheme conveys http/https, matching the boto3 store's dropping of is_secure); Azure storage_account/resource_group (previously unreachable — cloudbridge synthesizes a UUID-based storage-account name without them), region, access_token; GCP inline credentials_dict + region.
  • New provider: openstack: native Swift via Keystone v3 (password or application-credential auth), giving swift deployments a migration target.
  • User-defined storage templates: "cloud" is now an ObjectStoreTemplateType with full pydantic template/configuration models (provider, per-provider auth, bucket, connection, transfer), a production catalog example, and the client schema/type entries.

Tests

  • 20+ new unit tests (config parsing incl. direction-prefixed transfer options, provider config mapping via patched factory, upload/download config pass-through, direct download)
  • Env-gated live tests: AWS multipart (12 MiB over a 5 MiB threshold → 3 parts) and OpenStack Swift (GALAXY_TEST_OS_*), both exercising a SHA256-checked big-file roundtrip
  • New MinIO-backed integration test (test/integration/objectstore/test_cloud_objectstore.py) — deterministic multipart coverage in CI with no cloud credentials
  • The direct-download redirect integration suite (test_direct_download_redirect.py) now also runs against the cloud store (same five web-layer scenarios, own MinIO container)

Dependencies: cloudbridge>=4.3.0; the split 3.2.0/4.0.x pins collapse to one now that cloudbridge supports Python 3.10+. Pin changes are limited to cloudbridge itself.

How to test

  • pytest test/unit/objectstore/test_objectstore.py -k cloud (63 tests in the file pass overall)
  • pytest test/integration/objectstore/test_cloud_objectstore.py (docker)
  • Live: GALAXY_TEST_AWS_* pytest test/unit/objectstore/test_objectstore.py -k cloudbridge_store_multipart
  • Validated locally against MinIO: bucket auto-creation, full caching-store lifecycle incl. presigned-URL download, 12 MiB multipart roundtrip (SHA256-checked)

Deferred follow-ups

  • s3.py deprecation announcement + migration docs (aws_s3 → boto3/cloud; swift/generic_s3 → boto3+endpoint_url or cloud+openstack)

nuwang added 20 commits July 13, 2026 00:23
…entials

The aws branch of Cloud.parse_xml checked `"region" in auth_element` —
membership on an XML Element iterates child elements, so the check was
always false and a configured region was silently dropped (the
downstream connection code at _get_connection already handled a region
correctly when present).

The google branch called os.path.isfile() on the credentials_file value
before checking it for None, so a missing attribute produced a TypeError
instead of the intended missing-configuration error.
Both S3 stores set the `cloud = True` class attribute (surfaced through
badge serialization); the cloudbridge store left it at the ConcreteObjectStore
default of False.
Both push paths did an objects.get() to test existence and then a second
get() to fetch the same object for upload. Get-or-create once and upload
on the returned object.
A new <transfer> block (multipart_threshold, multipart_chunksize,
max_concurrency - byte sizes, boto3-store naming) maps onto
cloudbridge's per-call UploadConfig. multipart_chunksize below the
5 MiB provider minimum is rejected at construction time rather than
failing on first large upload.

Only upload tuning is exposed: cloudbridge has no download-side
transfer configuration (downloads stream in fixed chunks).
upload()/upload_from_file() now receive the transfer options parsed from
the <transfer> block; with nothing configured they receive config=None
and cloudbridge's own multipart defaults apply. cloudbridge handles the
multipart mechanics transparently above the threshold.
The enable_direct_download plumbing is generic (ConcreteObjectStore
dispatches to _get_object_url), but the cloud store's _get_object_url
accepted content_disposition/content_type and dropped them. Forward
both to cloudbridge's generate_url so redirected downloads carry the
forced filename and MIME type (requires cloudbridge >= 4.2.1; on
OpenStack Swift only the filename portion is honored).
…tore

The store surfaced only a fraction of what each cloudbridge provider
accepts; several missing options are required in real deployments:

- aws: session_token (STS/temporary credentials) on <auth>, and a
  <connection> block with endpoint_url (S3-compatible endpoints such as
  MinIO/Ceph RGW - the scheme conveys http/https, so no is_secure,
  matching the boto3 store), validate_certs and signature_version.
- azure: storage_account and resource_group - without these cloudbridge
  synthesizes a UUID-based storage-account name, so an existing storage
  account was unreachable. Also region and access_token (with a token
  the service-principal quartet becomes optional).
- google: credentials_dict as an inline alternative to credentials_file
  (YAML configurations; exactly one of the two is required) and region.

Compute-only cloudbridge options (zones, VM settings, ec2_*) are
deliberately not exposed. Unset options are omitted from the provider
config so cloudbridge's own env-var fallbacks still apply.
Native Swift object storage via cloudbridge's OpenStack provider,
giving swift/generic_s3 deployments a migration target ahead of the
legacy boto2 store's deprecation. Authentication supports Keystone v3
password auth (username/password/project_name) or an application
credential, plus auth_url (required), region and domain names.

Requires the cloudbridge[openstack] extras in Galaxy's environment,
matching how the azure/google providers expect their SDKs.
verify_caching_object_store_functionality's "bigger file" check never
actually uploaded the big file - it wrote big_file.bytes but passed
hello_path to update_from_file, and asserted nothing. Upload the right
file and assert existence and size.

A new verify_big_file_storage_roundtrip helper stores a file large
enough to cross the multipart threshold and verifies a bit-for-bit
SHA256 roundtrip through the remote store after cache resets. Two new
env-gated tests exercise it with a 12 MiB file over a 5 MiB
threshold/chunksize: against real S3 (cloudbridge maps the config onto
boto3's TransferConfig) and against real Swift (cloudbridge's generic
clone-pool multipart driver, gated on GALAXY_TEST_OS_*).
Runs the cloudbridge store (provider=aws + endpoint_url) against MinIO
in docker - no cloud credentials needed - with the multipart threshold
at the 5 MiB provider minimum, so the tool-test datasets and an
explicit 12 MiB roundtrip exercise the multipart upload path in CI.

Verified locally against MinIO: bucket auto-creation, the full caching
object store lifecycle including presigned-URL download, and the
12 MiB multipart roundtrip (SHA256-checked).
Covers direct download, STS session tokens, S3-compatible endpoints,
multipart transfer tuning, Azure storage_account/resource_group/
access_token, GCP inline credentials, and the new openstack provider.
4.2.1 supports Python 3.10+ (so the split 3.2.0/4.0.x pins collapse to
one) and provides the multipart UploadConfig and generate_url
response-header parameters the cloud object store now uses. Pins
regenerated with ./lib/galaxy/dependencies/update.sh -p cloudbridge,
which also picked up patch-level drift in a few unrelated packages.

Also add cloudbridge to the objectstore package's test extra.
…tore

Downloads now go through cloudbridge's download_to_file, which fetches
objects above the transfer threshold as ranged reads in parallel -
mirroring what boto3's TransferManager gives the boto3 store - instead
of a single-connection stream. The axel fallback is dropped: parallel
downloads are now native, and the boto3 store never used axel either.

The <transfer> options gain optional upload_/download_ prefixes to tune
one direction only (bare keys apply to both), matching the boto3
store's convention, and are validated accordingly: the 5 MiB part floor
is an upload constraint, so download chunk sizes below it are legal.

cloudbridge 4.3.0 also renames UploadConfig to TransferConfig now that
the same knobs drive both directions.
Adds "cloud" to ObjectStoreTemplateType with pydantic template and
configuration models covering the store's full surface: provider
(aws/azure/google/openstack), a flat per-provider auth model (which
fields apply depends on the provider - the store validates the
combination at construction), bucket, connection and transfer options.
Includes a production catalog example for an S3 bucket via CloudBridge
and the regenerated client API schema plus the template type's
icon/message entry in the client.
cloudbridge's objects.list() returns a single page (default 50 items),
and the store iterated only that first page when deleting all objects
under a prefix and when downloading a directory into the cache - a
dataset with more extra files than the page size was deleted or
downloaded incompletely. Use objects.iter(), which pages through the
complete result set on every provider.

The directory existence check keeps list(): any match at all puts at
least one object on the first page.
The referenced issue was a typo (CloudVE/cloudbridge#135 is unrelated;
the credential-check discussion is galaxyproject#120). The reasoning still holds -
cloudbridge's authenticate() is compute-scoped, so a least-privilege
bucket-only credential would fail it - and since the bucket lookup in
_initialize now exercises the credentials with a storage-scoped call,
bad credentials fail fast at startup anyway.
…store too

The object store config becomes a class attribute so the cloud
(cloudbridge) store runs the same five web-layer scenarios the boto3
store does, against its own MinIO container: /download 302s to a
presigned URL carrying the response content disposition, the URL is
fetchable directly from the store with no cache pull, HEAD serves
metadata without a redirect, and /display keeps streaming.

Also pin down enable_direct_download on the cloud store's to_model
serialization, matching the boto3 parse test.
ObjectStoreTypeSpan.vue indexes an exhaustive record over the template
type enum, so it needs a message for the new cloud type. The paging
fake's SimpleNamespace provider gets a cast for mypy - only the
result-limit config is consulted by ClientPagedResultList.
The paragraph predates this branch: the cloud store now has parallel
multipart transfers in both directions with per-direction tuning, and
handles S3-compatible endpoints via endpoint_url.
@nuwang
nuwang force-pushed the cloud-objectstore-parity branch from f48bcb0 to fc6dcbd Compare July 12, 2026 19:11
@nuwang
nuwang marked this pull request as ready for review July 12, 2026 19:17
@mvdbeek
mvdbeek requested a review from jmchilton July 14, 2026 14:39
Comment on lines +5 to +7
Stores Galaxy datasets in an Amazon S3 bucket through the
provider-agnostic cloud object store. Files larger than the multipart
threshold transfer as parts in parallel, in both directions.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a user facing file - to keep it inline with the other examples I would not include such technical language.

@jmchilton jmchilton left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To bring it to parity I'd also recommend updating docs/ around this. All the other templates I think have admin-facing documentation, schema docs, etc... But I don't think it is strictly required. Also just a warning - the template objects that can be used to defined user-defined object stores need to essentially maintain backward compatibility indefinitely because those configs are defined in the database. Therefore, I tried to be a little conservative about what object stores and what configs I expose in those places based on that (admittedly I may have not used such restraint with the boto3 implementation though). So I just want to make sure you're comfortable with cloudbridge supporting this interface defined in that model layer indefinitely. I don't see any obvious problems - just letting you know that I'm uneasy every time this interface expands and this is a substantial expansion.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

Status: Needs Review

Development

Successfully merging this pull request may close these issues.

2 participants